(12) 



UK Patent Application ,fi,QB ^,,,2 300 535 ,,3, A 



(43) Date of A Publication 06.11.1998 



(21) Application No 9507385.4 

(22) Date of Filing 10.04.1995 



(71) Applicant{s) 

Quantet Umfted 

(Incorporated In the Unhed Kingdom) 

Pear Tree Lane, NEWBURY. Berks, RG13 2LT, 
UnHed Kingdom 

(72) lnventor(s) 

Timothy John Beckwrth 

(74) Agent and/or Address for Service 
R G C Jenkins & Co 

26 Caxton Street LONDON, SW1H ORJ, 
United Kingdom 



(51) INTCL* 

H04N 5/262 , G11B 27/031 27/34 

(52) UK CL (Edition O) 

H4F FD12M FD12X FD2A FD2B FD30K FD31K FGG 

G5R RB789 RB81 

H4T TBLX T128 T144 T149 T150 

(56) Documents Cited 
US 5237648 A 

(58) Field of Search 

UK CL (Edition N ) H4F FGG FGS , H4T TBAD TELA 
TBLC TBLM TBLX 

INT CL® G11B 27/022 27/024 27/026 27/028 27/031 
27/032 27/034 27/34 , H04N 5/262 5/265 



(54) Video processing system for displaying and editing video clips 

(57) An electronic video processing system comprises a store for storing data defining plural video clips, and 
a processor for selecting from the stored video clips a plurality of clips and for selectively combining data 
defining the selected clips to form data defining a video story. A plurality of representative frames 55 
respectively representing selected video edits is displayable to portray a portion of the video story together 
with coloured strips 62 which indicate the position within a displayed clip of the respective frames. Also, for 
each selected clip, a frame representative thereof is displayable together with a pair of smaller frames 73, 74 
derived from the first and last frames of the clip. 



t5 



2ii 



73 
7t 



71- 



72 



4-73 
7t 



55 



35a 



36a 



— 



35b-^ ^ „ 36b^l2 
rh35 

36 



Li 

70 

X9 



'-'35ci-' 
35d 
35e 



70 



55 



62' 



JI 



50 



I., 



1,7 




r 






52 


r 






^51 


55 




r 


^50 


r 




^60 


r 


r 




-35c 




r 





w 

CO 



Ol 
Ca) 
CJI 



FIG.2. 



36d' ^^366 



At least one drawing originally filed was informal and the print reproduced here is taken from a later filed formal copy. 
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This invention relates to an electronic video processing system. 
In our co-pending British application No. 920SS03.7 (publication No. 2266037) filed 
on 13 March 1992 and equivalent applications at the European Patent Office No. 
93301889.7 (publication No. 0560624) filed on 12 March 1993, in the United States of 
America No. 08/030823 filed on 12 March 1993 and m Japan No. 55786/1993 filed on 
15 March 1993, the contwits of which are herein incorporated by reference, there is 
described an electronic video system for displaying a plurality of video clips having 
storing means for storing a multiplicity of video fi-ames, a processor connected with 
said storing means, user operable input means connected with the processor and 
serving to enable the processor to fetch from and store in the storing means data of 
video frames comprising any one of said video clips, a display store connected to the 
processor for storing video clip fi-ames brought by the processor fi-om or supplied by 
the processor to the storing means and a monitor connected with the display store for 
displaying frames of video data brought by the processor to the display store. 
The co-pending application and its equivalents also describes the display monitor as 
displaying and combining video clips in a variety of ways by user operable input means, 
specifically a pen and tablet, for example, by splicing, creation of dissolves etc., and in 
each case previewing of the combined clip is afforded so that the operator need not 
commit himself to a particular combination until he is completely satisfied with the 
form of the combined clip. 

Whilst the system described in our co-pending applications referred to has proved to 
be commercially widely acceptable, one way in which it is capable of improvement has 
been identified. In using the described apparatus to assemble a story by combining and 
editing clips from a sequence of clips, the need to change edit points in the sequence or 
the order of clips in the sequence occurs frequently and involves scrolling through the 
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sequence of clips until the region of the sequence is displayed which contains the frame 
or frames at which the editing is to take place or at which the order of the clips is to be 
changed. This is a time consuming process which is addressed and speeded up by the 
present invention. 

The present invention consists in an electronic video processing system for displaying 
video clips, comprising storing means for storing a multiplicity of video frames, a 
processor connected with said storing means, user operable input means connected 
with the processor and serving to enable the processor to fetch from and store in the 
storing means data of video frames comprising any one of said video clips, a display 
store connected to the processor for storing video clip frames brought by the processor 
from or supplied by the processor to the storing means and a monitor connected with 
the display store for displaying frames of video data brought by the processor to the 
display store, characterised in that the processor is enabled by operation of said user 
operable means to display on the monitor a video clip comprising frames brought by 
the processor from said storing means to the display store, to select any frame of said 
displayed clip and separately to display said selected frame together with indicating 
means which indicate the position in the displayed clip of the selected frame. 
Advantageously the user operable means by causing the processor to display on the 
display monitor a cursor and selecting therewith the indicating means is capable of 
moving the cursor relatively to the indicating means thereby both to replace the 
selected frame with any other frame of the displayed clip and to adjust the indicating 
means to indicate the location of the frame which replaced the selected frame. 
Suitably, the indicating means is a distinctively marked narrow strip displayed on the 
monitor alongside the separately displayed frame. 
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By a further feature of the invention, the user operable means by causing the processor 
to bring into the display store clips for display and to display said clips in sequence on 
the monitor can select for each clip a representative frame for display on the monitor 
separately from the associated clip and together with an associated indicating means 
which indicates the position of the representative frame in its associated clip. 
Preferably, the displayed representative frames and then- associated indicating means 
are displayed as a sequence of frames. 

In one form of the invention the processor on instruction from the user operable means 
displays in an area of the monitor separate from the displayed sequence of 
representative frames a copy of said sequence together with, for each frame of the 
sequence, a pair of reduced sized frames displaying respectively the start and end 
frames of the clip of the associated representative frame. 

Preferably, the sequence of representative frames is displayed as a column of frames 
with the respective indicating means displayed as narrow, distinctively marked strips 
between pairs of the representative frames and said reduced size frames form a column 
of image frames alongside the column of representative frames each pair of reduced 
sized images occupying the vertical height of the associated representative frame. 
Suitably, the user operable means by causing the processor to display a cursor on the 
monitor can select with said cursor any one of the reduced sized images and by 
movement of the cursor relatively to the selected image can cause the selected image 
to sequence through the visible images of the corresponding clip and also through 
images beyond the visible images of the clip not used in the join between the clip and 
an adjoining clip. 

The invention will now be described, by way of example, with reference to the 
accompanying drawings, in which:- 
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Figure 1 is a block diagram of a system capable of performing the present invention, 
and 

Figure 2 illustrates a screen of a display monitor which is part of the system of Figure 
1. 

Referring first to Figure 1, which corresponds closely with Figure 2 of our co-pending 
application referred to, there is shown an electronic video processing system 10 in 
which video clip data is stored in a video store 11. The video data is input to the video 
store 1 1 via an input selector 12 which includes both an analog input 13 and a digital 
input 14 to enable data to be input in either analog or digital format from an off-line 
bulk storage device or library (not shown) such as a VTR. The video clip data is 
stored in digital form in the video store 1 1 and the input selector therefore comprises a 
suitable converter for converting the incoming clip data into the required format. The 
input selector 12 is connected to the video store 11 via two bi-directional data paths 
15, 16 each capable of conveying a full frame of data at at least video rate to and fi^om 
the video store 1 1 . 

The video store as described in our co-pending application referred to, comprises at 
least two disc packs, each having two disc drive devices in parallel. Each pack 
requires two disc drives in order to achieve the necessary bandwidth since each disc is 
only able to receive or output data at approximately half broadcast standard video rate. 
The video store as more fully described in our co-pending application is a truly random 
access frame store which thereby enables frames from different video clips to be 
accessed in a random order for output therefrom. The provision of two bi-directional 
paths 15, 16 enables two frames of data to be simultaneously written to or read from 
the store 1 1 or for one frame to be written to while another is read from the store 11. 
This enables a system 10 of great flexibility to be provided. 
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The system 10 further comprises a display store 22 which is comiected via two bi- 
directional busses 23, 24 to the input selector 12. The display store 22 comprises a 
large scratch pad store for storing data relating to several frames of video and a display 
processor 21 for processing the frame data therein to produce respective frames of 
reduced size for concurrent display at diflFerent portions of a monitor 25, as will be 
described in greater detail hereinafter. 

A video clip may be read out from the video store 1 1 and written directly to the display 
store 22 at video rate for display on the monitor 25 either at full size or at half, quarter 
or eighth size. Alternatively video clips may be transferred from a bulk storage library 
(not shown) via the input selector 12 to the display store 22. Data from the display 
store 22 is read by an output selector 26 and output to the monitor 25 for display. 
This enables the user to preview one or more video clips or to identify video clips 
stored off-line in a bulk store library (not shown) for transfer to the video store 11. 
The output selector 26 also comprises both analog and digital outputs 26a, 26b, similar 
to the analog and digital inputs 13, 14 of the input selector 12, to enable video clip 
data to be output in either analog or digital format to the bulk storage device for 
example. 

In an editing mode of operation data relating to a plurality of video clips is read out 
from the video store 1 1 a frame at a time to the display store 22 where the data is 
processed before being stored so as to enable several fi^mes to be displayed 
simultaneously at different, overiapping, or shared, portions of the monitor 25, as 
described in greater detail in our co-pending application referred to. The processed 
data is output from the display store in raster order for display on the monitor 25. The 
video store has associated with it a video processor 27 and an. effects processor 28. 
The video processor 27 is arranged to perform such operations as generating a keying 
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signal, modifying colour, changing texture, and the effects processor 28 generates 
spatial effects such as changes of size, position and spin to one or more frames of a 
video clip, which operations and effects are all, per se, well known. Frames modified 
by the video processor 27 and effects processor 28 are supplied or returned to the 
display store 22 for display on the monitor 25. 

The display store 22 also has associated with it a graphics processor 29 which enables 
a user to paint into each frame as described in our British Patent No. 2,089,625 and/or 
to create a key or stencil as described in our British Patent No. 2,113,950. The 
graphics processor 29 is responsive to a user operable input device such as a pen or 
stylus and touch tablet combination 30, as indeed are the video processor 27, the 
effects processor 28 and the display store processor 21. 

During editing the video processor in the display store 22 is arranged to generate from 
the input video clips a display that represents an editing environment such as the 
display shown in Figure 2 of the accompanying drawings. As shown in Figure 2, the 
display is divided into three distinct working areas 31, 32, 33 by three so-called reel 
bars 34, 35 and 36, 

Each reel bar 34 to 36 includes a marker 45 to 47 which is movable along the reel bar 
to provide an indication of the position of the displayed frames in relation to the clip as 
a whole and each displayed frame includes at its bottom right hand comer an 
information box, for example 48 to 52, to provide a further indication of the positions 
of the displayed frames in the video clips. The information in each box may for 
example be the number, i.e. position, of the frame in the clip or it may be time codes 
generated by a VTR when the frames were originally recorded. 
A cursor 57 which is generated by selection by the pen and tablet of an icon in a menu 
of facilities and is moved by movement of the pen on the touch tablet is used to control 
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many operations perfonned by the system 10. The cursor is displayed by selection 
from a menu displayed on the monitor. One function vMch the cursor S7 can be used 
to control is that of selecting which frames from a selected clip are displayed in the 
working areas 3 1 to 33. The reel bars 34 to 36 eadi have associated direction control 
icons 34a to e, 3Sa to e and 36a to e. The reel bars, the control icons and the cursor 
are ail generated by the processor 21 within the display store 22. Placing the cursor 57 
over the icon 36a for example and pressing the stylus firmly on the touch tablet will 
cause the clip displayed in work area 33 to scroll up whilst selecting icon 36b will 
cause the clip to scroll up to an event, such as a join, previously defined in the clip. 
Similar operations are performed by selection of the other icons as will be readily 
apparent to those possessed of the appropriate skills. When a scrolling icon is selected 
data relating to the appropriate video frames in the appropriate clip are read from the 
video store 1 1 to the display store 22. The data is processed to reduce the size of each 
frame and the processed data is then stored at locations in the store 22 corresponding 
to the position of the reduced frame on the monitor 25. 

Video clips such as clip 60 to be used in compiling a story are loaded from the bulk 
store (not shown) via the input selector 12 to the video store 11. One of the bi- 
directional paths, e.g. 15 is used to transfer the incoming video clip data to the video 
store 11 and this leaves the other path, e.g. path 16 free to transfer data relating to 
another video clip of the story. From the store 1 1, the data of the clip of the story can 
be read by the processor of the display store 22 into that store for display of the video 
on the monitor. 

Once the desired clip or clips have been loaded into the video store 1 1 and selected 
frames thereof displayed on the monitor 25 the user can manipulate the clips as now 
described in accordance with the present invention. 



After loading of the clip 60 from the store 1 1, alone or in sequence with other clips to 
be used in the story, into the display store 22 and displaying it in area 33 of the 
monitor, the pen and tablet or other suitable manually operable device in known 
manner causes the appropriate mode of operation of the display store processor to be 
seleaed for performing the present invention. 

The user first seleas by means of the pen and tablet 30 and the cursor S7 a fi^e, say 
frame SS, of the clip 60 as a fiame which is representative of the action portrayed by 
the clip 60. When the cursor is located over the fi^e, the pen is pressed down on the 
tablet and such pressure attaches the cursor to the frame 50. By movement of the 
cursor, means of by dragging of the pen on the tablet, to the working area 32 a copy of 
the frame SS appears in the area 32 together with an indicating means 61 in the form of 
a distinctively coloured strip 62 extending alongside the bottom edge of the frame from 
the left hand end thereof The length of the strip is an indication of the location of the 
frame 55 in its associated clip. Thus the shorter the strip the nearer the frame 55 is 
located to the beginning of the clip and the longer the strip the nearer the frame 55 is 
located to the end of the clip. 

The user can now locate the cursor by means of the pen over the right hand end of the 
strip and by then applying pressure to the pen and moving the pen on the tablet in the 
direction of the strip, the length of the strip can be varied. In changing the length of 
the strip the frame 55 is changed to a different frame of the clip which is located at a 
point in the clip which depends on the new length of the clip. By moving the cursor in 
this manner along the direction of the strip, the entire clip 60 can be sequenced and the 
image displayed can be changed to that of any other frame of the clip by lifting the pen 
when the desired frame is displayed. 
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The procedure is then repeated for further clips which are to be employed in the story 
to the end of providing in the working area 32 a column 70 of frames each with its 
own indicating strip and respectively representing the original clip 60 and the fiuther 
clips. The further clips can be brought into the display store eithCT one at a time by the 
display processor and displayed to pro^dde the corresponding frame for the column 70 
or they can be brought successively into the display store and atuched one to the next 
so that the entire length of the clips forming the story can be sequentially displayed 
using the scrolling facility mentioned earlier. 

At this juncture the clips are not necessarily in the order required for the story. To re- 
arrange the order of the clips, the pen and tablet is manipulated to bring the cursor 57 
over one of the images in the column 70 the position of the associated clip of which 
requires to be changed. The cursor is then attached to this image by pressing down the 
pen on the tablet and can then be caused to drag the attached image by movement of 
the pen over the tablet until the cursor is located over the junction between images of 
the colunui 70 at which the clip of the moved image is to be introduced. Release of the 
pressure on the pen then interposes the representative image of the clip between the 
images of the column at which the cursor is located. This operation is repeated as 
necessary on different images of the column 70 the associated clips of which have to be 
repositioned in the clip sequence to meet the story requirements. 
It will be appreciated that the column 70 of images representative of associated clips 
comprises, as it were, a table of contents of the story comprised by the assembled clips 
and enable the user quickly to access any part of the story of interest to him. 
By appropriate command, imparted by means of the pen and tablet, the processor 21 
can be ordered to display in the working area 3 1 a copy of the column 70 of clip 
representative images together with a further column 71 of equally sized, reduced size 
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images 72, suitably eighth size images, which extend alongside the column so that for 
each image of the column 70, there is a pair of images 73, 74 the combined vertical 
height of which occupies the vertical height of the corresponding image of the colunm 
70. The images 73, 74 are respectively the first and last of the visible images of the 
clip represented by the image of the column 70 adjacent the images 73, 74. By moving 
the cursor by means of the pen and tablet over one of the images 73, 74, e.g. the image 
74, and pressing the pen down on the tablet and holding it so pressed whilst moving it 
from side to side the images of the corresponding clip particularly the last part thereof, 
can be sequenced and the point of connection of the clip to the adjoining clip thereby 
changed simply by releasing the pressure on the pen when the desired image appears as 
the image 74. The image so appearing becomes the last visible image of the 
corresponding clip. During sequencing of the images to change the image 74, the 
corresponding image in the column 70 is changed to that displayed as image 74. By 
performing in like manner on one of the images 73, the first visible image of the 
corresponding clip can be changed. Where adjoining clips forming the story have been 
spliced as described in connection with Figure 4 of our co-pending application referred 
to and its equivalents filed in Europe, USA and Japan, the disclosures of which are 
incorporated herein by reference, the joins between clips may involve not using a frame 
or frames of the clips concerned at the joins. Such frames, usually referred to as 
"hidden" frames can be sequenced in series with the visible frames of the associated 
clip when the images 73 and 74 are subjected to change as described. This gives 
greater flexibility when deciding where the join between clips is to take place. 
Having now described the manner of performing the invention, it will be apparent that 
the invention provides a quick means of identifying a clip from the representative 
image of the clip displayed with the indicating means 61 and sequencing through it. 
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Also the column 70 of representative images together with their indicating means 
enables ready access to parts of a story formed by the series of clips without having to 
use the scrolling facility which by comparison is tedious. Further, a series of clips 
reduced to the column 70 can be rapidly re-ordered to suit the story dictates. Such 
advantages are obtained even before the additional facilities provided by the further 
column 71 of image 73 and 74 are available such faciUties affording the additional 
advanuges of initially showing the first and last visible frame of the corresponding clip, 
of being able to sequence through the end frames not only of the visible frames of the 
dip but also of those frames which adjoin the visible frames thus enabling the latter 
fi^mes to be employed in the story, and, of avoiding the use of previously visible 
frames at a newly detennined join. The determination of a new join between clips is 
further facilitated by the fact that as image sequencing is performed on the images 73 
and/or 74 the conesponding representative image of the associated clip changes so that 
the image appearing as image 73 or 74 depending on which of these is being sequenced 
also appears as the adjoining dip representative image rendering it easier to locate a 
desired point at which to make a new join because the size, and therefore the 
resolution, of the image in the column 70 is twice that of the corresponding small sized 



image. 
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1) An electronic video processing system for displaying video clips, comprising storing 
means for storing a multiplicity of video fi^es. a processor connected with said 
storing means, user operable input means connected with the processor and serving to 
enable the processor to fetch from and store in the storing means data of video frames 
comprising any one of said video clips, a display store conneaed to the processor for 
storing video clip frames brought by the processor from or supplied by the processor 
to the storing means and a monitor connected with the display store for displaying 
frames of video data brought by the processor to the display store, characterised in that 
the processor is enabled by operation of said user operable means to display on the 
monitor a video clip comprising frames brought from said storing means to the display 
store, to select any frame of said displayed clip and separately to display said selected 
frame together with indicating means which indicate the position in the displayed clip 
of the selected frame. 

2) A system as claimed in claim 1, characterised in that the user operable means by 
causing the processor to display on the display monitor a cursor and selecting 
therewith the indicating means is capable of moving the cursor relatively to the 
indicating means thereby both to replace the selected frame with any other frame of the 
displayed clip and to adjust the indicating means to indicate the location of the frame 
which replaced the selected frame. 

3) A system as claimed in claim 2, characterised in that the indicating means is an 
elongated strip and said cursor is moved lengthwise thereof to change the length of the 
strip. 
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4) A system as claimed in claim 1, claim 2 or claim 3, characterised in that the indicating 
means is a distinctively mariced narrow strip displayed on the monitor alongside the 
separately displayed frame. 

5) A system as claimed in any preceding claim, characterised in that the user operable 
means by causing the processor to bring into the display store clips for display and to 
display said clips in sequence on the monitor can select for each clip a representative 
frame for display on the monitor separately from the associated clip and together with 
an associated indicating means which indicates the position of the representative frame 
in its associated clip. 

6) A system as claimed in claim S, characterised in that the displayed representative 
frames and their associated indicating means are displayed as a sequence of frames. 

7) A system as claimed in claim 6, characterised in that the processor on instruction from 
the user operable means displays in an area of the monitor separate from the displayed 
sequence of representative frames a copy of said sequence together with, for each 
frame of the sequence, a pair of reduced sized frames displaying respectively the start 
and end frames of the clip of the associated representative frame. 

8) A system as claimed in claim 7, characterised in that the sequence of representative 
frames is displayed as a column of frames with the respective indicating means 
displayed as narrow, distinctively marked strips between pairs of the representative 
frames and said reduced size frames form a column of image frames alongside the 



column of representative frames each pair of reduced sized images occupying 
vertical height of the associated representative frame. 



A system as claimed in claim 8. characterised m that the user operable means by 
causing the processor to display a cursor on the monitor can select with said cursor 
any one of the reduced sized images and by movement of the cursor relatively to the 
selected image can cause the selected image to sequence through the visible images of 
the corresponding dip and also through images beyond the visible images of the clip 
not used in the join between the clip and an adjoining clip. 

A system as claimed in claim 9. characterised in that the user operable means is a pen 
and tablet and movement of said cursor is effected by holding the pen under pressure in 
contact with the tablet and moving the pen relatively to the tablet, whilst said pressure 
is maintained, selection of the image of the images being sequenced occurring when 
said pressure is relieved. 

A system as claimed in any preceding claim, characterised in that there are provided 
access means connecting the storing means with the display store capable of 
transferring video data at video rate between the display store and the storing means. 

A system as claimed in any preceding claim, characterised in that the processor is 
incorporated in the display store 
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